1 link tagged with all of: kafka + linkedin + metadata management + storage architecture
Click any tag below to further narrow down your results
Links
LinkedIn built Northguard and Xinfra to replace Kafka's core design after hitting limits at 32 trillion records per day. The new system decouples logs from replication units, distributes metadata across independent state machines, and lets new brokers serve traffic without copying historical data.
- Kafka's partition couples ordering, replication, and consumer parallelism into one heavyweight unit; moving or repairing partitions at LinkedIn's scale wastes network and disk bandwidth. Northguard uses smaller segments with independent replica assignments, so adding capacity doesn't require copying old data.
- The centralized metadata controller became a bottleneck managing millions of partition replicas. Northguard shards metadata across 128+ coordinators using a hash ring, with each vnode managing its own Raft state machine for topic, range, and segment metadata.
- Northguard can enforce fsync on all replicas before acknowledging writes while meeting LinkedIn's performance SLOs—a durability guarantee that Kafka's default configuration doesn't require because it relies on replication instead.